{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "4facdf7f-680e-4d28-908b-2b8408e2a741",
      "metadata": {},
      "source": [
        "# How to use multimodal prompts\n",
        "\n",
        "Here we demonstrate how to use prompt templates to format multimodal inputs to models. \n",
        "\n",
        "In this example we will ask a model to describe an image.\n",
        "\n",
        ":::info Prerequisites\n",
        "\n",
        "This guide assumes familiarity with the following concepts:\n",
        "\n",
        "- [Chat models](/docs/concepts/chat_models)\n",
        "- [LangChain Tools](/docs/concepts/tools)\n",
        "\n",
        ":::\n",
        "\n",
        "```{=mdx}\n",
        "import Npm2Yarn from \"@theme/Npm2Yarn\"\n",
        "\n",
        "<Npm2Yarn>\n",
        "  axios @langchain/openai @langchain/core\n",
        "</Npm2Yarn>\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "0d9fd81a-b7f0-445a-8e3d-cfc2d31fdd59",
      "metadata": {},
      "outputs": [],
      "source": [
        "import axios from \"axios\";\n",
        "\n",
        "const imageUrl = \"https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg\";\n",
        "const axiosRes = await axios.get(imageUrl, { responseType: \"arraybuffer\" });\n",
        "const base64 = btoa(\n",
        "  new Uint8Array(axiosRes.data).reduce(\n",
        "    (data, byte) => data + String.fromCharCode(byte),\n",
        "    ''\n",
        "  )\n",
        ");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "2671f995",
      "metadata": {},
      "outputs": [],
      "source": [
        "import { ChatPromptTemplate } from \"@langchain/core/prompts\";\n",
        "import { ChatOpenAI } from \"@langchain/openai\";\n",
        "\n",
        "const model = new ChatOpenAI({ model: \"gpt-4o\" })"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "4ee35e4f",
      "metadata": {},
      "outputs": [],
      "source": [
        "const prompt = ChatPromptTemplate.fromMessages(\n",
        "    [\n",
        "        [\"system\", \"Describe the image provided\"],\n",
        "        [\n",
        "            \"user\",\n",
        "            [{ type: \"image_url\", image_url: \"data:image/jpeg;base64,{base64}\" }],\n",
        "        ]\n",
        "    ]\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "id": "089f75c2",
      "metadata": {},
      "outputs": [],
      "source": [
        "const chain = prompt.pipe(model);"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "id": "02744b06",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The image depicts a scenic outdoor landscape featuring a wooden boardwalk path extending forward through a large field of green grass and vegetation. On either side of the path, the grass is lush and vibrant, with a variety of bushes and low shrubs visible as well. The sky overhead is expansive and mostly clear, adorned with soft, wispy clouds, illuminated by the light giving a warm and serene ambiance. In the distant background, there are clusters of trees and additional foliage, suggesting a natural and tranquil setting, ideal for a peaceful walk or nature exploration.\n"
          ]
        }
      ],
      "source": [
        "const response = await chain.invoke({ base64 })\n",
        "console.log(response.content)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e9b9ebf6",
      "metadata": {},
      "source": [
        "We can also pass in multiple images."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "id": "02190ee3",
      "metadata": {},
      "outputs": [],
      "source": [
        "const promptWithMultipleImages = ChatPromptTemplate.fromMessages(\n",
        "    [\n",
        "        [\"system\", \"compare the two pictures provided\"],\n",
        "        [\n",
        "            \"user\",\n",
        "            [\n",
        "                {\n",
        "                    \"type\": \"image_url\",\n",
        "                    \"image_url\": \"data:image/jpeg;base64,{imageData1}\",\n",
        "                },\n",
        "                {\n",
        "                    \"type\": \"image_url\",\n",
        "                    \"image_url\": \"data:image/jpeg;base64,{imageData2}\",\n",
        "                },\n",
        "            ],\n",
        "        ],\n",
        "    ]\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "id": "42af057b",
      "metadata": {},
      "outputs": [],
      "source": [
        "const chainWithMultipleImages = promptWithMultipleImages.pipe(model);"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "id": "513abe00",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The two images provided are identical. Both show a wooden boardwalk path extending into a grassy field under a blue sky with scattered clouds. The scenery includes green shrubs and trees in the background, with a bright and clear sky above.\n"
          ]
        }
      ],
      "source": [
        "const res = await chainWithMultipleImages.invoke({ imageData1: base64, imageData2: base64 })\n",
        "console.log(res.content)"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "TypeScript",
      "language": "typescript",
      "name": "tslab"
    },
    "language_info": {
      "file_extension": ".ts",
      "mimetype": "text/x.typescript",
      "name": "typescript",
      "nb_converter": "script",
      "pygments_lexer": "typescript",
      "version": "5.3.3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}
